Add /files/dags to PYTHONPATH in Breeze#61436
Add /files/dags to PYTHONPATH in Breeze#61436jason810496 wants to merge 1 commit intoapache:mainfrom
Conversation
Fix path Fix comments
|
Hmm. I think that should not be the case - at least by default when Airflow imports Dags (whether it's in Triggerer, worker or Dag Processor), it should add bundle root to the PYTHONPATH automatically - and we should avoid adding it externally- if we see a problem that it is not added, we should fix it. @jedcunningham @ephraimbuddy - am I right ? |
|
Correct (edit: mostly). In fact, triggers from Dag files was pretty broken before anyways, so we intentionally removed that ability in AF3. edit:
Yes for worker and dag processor. But not for the triggerer - the single process architecture means you can't do it reliably. |
|
See #48603. |
|
Thanks Jarek and Jed for sharing the context. I think there two parts we pointed out here
Agreed. For first part: Even though it's pointed out as bad practice as From the user perspective, I would expect my triggers defined in Dag files could be imported by Triggerer, but the Triggerer doesn't respect
We are able to restore the importing triggers from Dag files by adding airflow/airflow-core/src/airflow/jobs/triggerer_job_runner.py Lines 885 to 890 in fe0633d
Then this level of problem will be second part: Unless we refactor Triggerer to run the triggers by |
|
Thanks again for sharing the context! I will workaround with |
Don't focus as much on my PR description, but the changes in the PR. It's not just "best practice" not to in Airflow 3, we intentionally removed the ability to have them in a dag file. Also, don't focus on "dags folder" - you must have a multi-bundle mental model now. The "dags folder" config is simply a default for familiarity sake. It is completely valid to have a deployment that does not use |
Why
The default dags path in Breeze will be
/files/dags, but/files/dagsis not inPYTHONPATH, which cause the triggers defined in Dag file are not able to import by Triggerer, and have to workaround by copying Dag file to some directories that are inPYTHONPATH. Pointed out in: #58676 (comment)What
We should add
/files/dagstoPYTHONPATHin Breeze in case the triggers are defined in dags, and cause "unable to import trigger" during Triggerer runtime.Before:
/files/dagsis not inPYTHONPATHAfter:
/files/dagsis now inPYTHONPATH